Monitoring web site content

ABSTRACT

Systems and methods for monitoring Web page content associated with processing a resource request are provided. A client computing device generates a sample image corresponding to a set of resources rendered in response to a resource request. A processing component, such as an image analysis component, then compares the sample image with an averaged reference image to identify a compared image. The averaged reference image is determined from averaging a pixel intensity value for each pixel in a first reference image with a pixel intensity value for each pixel in a second reference image. These first and second reference images both correspond to the same set of requested resources. The processing component then weights the compared image to produce a weighted compared image and determines whether a sum of the intensity values for each pixel in the weighted compared image exceeds a threshold. Aspects of systems and methods for generating an alert message if the threshold is exceeded are also provided.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.13/006,281, entitled “MONITORING WEB SITE CONTENT” and filed on Jan. 13,2011, which in turn is a continuation of U.S. patent application Ser.No. 12/410,251, now U.S. Pat. No. 7,917,618, entitled “MONITORING WEBSITE CONTENT” and filed on Mar. 24, 2009, the disclosures of which areincorporated herein by reference.

BACKGROUND

Generally described, computing devices and communication networks may beutilized to exchange information. In a common application, a computingdevice may request content from another computing device via acommunication network. For example, a user at a personal computingdevice may utilize a browser application to request a web page from aserver computing device via the Internet. In such embodiments, the usercomputing device may be referred to as a client computing device and theserver computing device may be referred to as a content provider.

Content providers are generally motivated to provide requested contentto client computing devices often with consideration of efficienttransmission of the requested content to the client computing deviceand/or consideration of a cost associated with the transmission of thecontent. Additionally, the content requested by the client computingdevices may have a number of components, which may require furtherconsideration of latencies associated with delivery of the individualcomponents as well as the originally requested content as a whole. Evenfurther, content providers are also desirous of providing contentwithout errors or with minimal errors such that the content renders asexpected.

With reference to an illustrative example, a requested Web page, ororiginal content, may be associated with a number of additionalresources, such as images or videos, which are to be displayed with theWeb page. In one specific embodiment, the additional resources of theWeb page are identified by a number of embedded resource identifiers,such as uniform resource locators (“URLs”). In turn, software on theclient computing devices, such as a browser application, typicallyprocesses embedded resource identifiers to generate requests for thecontent. Often the resource identifiers associated with the embeddedresource reference a computing device associated with the contentprovider such that the client computing device would transmit therequest for the additional resources to the referenced computingdevices. Accordingly, in order to satisfy a content request, the contentprovider(s) (or any service provider on behalf of the contentprovider(s)) would provide client computing devices data associated withthe Web page and/or data associated with the embedded resources.

Traditionally, a number of methodologies exist which measure theperformance associated with the exchange of data and the functioning ofthe underlying software in the environment described above. For example,some methodologies provide for limited measurement of performancemetrics associated with network side processing of a content request.Other methodologies allow for limited measurement of performance metricsassociated with the content request measured from the browser side. Fromeither or both of the foregoing methodologies, implicit softwarefailures can be monitored by observing software performance for abnormalbehavior (e.g., latency monitoring). Alternatively, other methodologiesmeasure implicit software failures by analyzing business metrics such asitem order rates or number of dropped item orders. Still further, othermethodologies provide for monitoring explicit software failures, such asHTTP status codes, fatal classifications from RTLS, CX fatals, and thelike.

BRIEF DESCRIPTION OF THE DRAWINGS

Many of the attendant advantages and aspects of the present disclosurewill become more readily appreciated as the same become betterunderstood by reference to the following detailed description, whentaken in conjunction with the accompanying drawings, wherein:

FIG. 1 is a block diagram illustrative of a Web site monitoring systemincluding a number of client computing devices, a content provider, anda processing device;

FIG. 2 is a block diagram of the Web site monitoring system of FIG. 1illustrating the process of monitoring and fulfilling resource requests;

FIG. 3 is a block diagram of the Web site monitoring system of FIG. 1illustrating the process of providing image data corresponding torendered Web page content from a client computing device and processingthe image data at a processing device;

FIG. 4 is a flowchart illustrative of an image processing routineimplemented by the Web site monitoring system for processing referenceimage data associated with a set of requested resources rendered by theclient computing device;

FIG. 5 is a flowchart illustrative of an image monitoring routineimplemented by an image analysis component of the Web site monitoringsystem for monitoring performance associated with the set of requestedresources rendered in accordance with a further request for the set ofresources;

FIG. 6A is an illustrative user interface displaying an expected imagecorresponding to a set of requested resources;

FIG. 6B is an illustrative user interface displaying a sample imagecorresponding to the set of requested resources rendered in accordancewith a further request for the set of resources; and

FIG. 6C is an illustrative user interface displaying an area in whichthe averaged reference image and the sample image differ.

DETAILED DESCRIPTION

Generally described, the present disclosure is directed to monitoringthe performance associated with rendering Web site content at clientcomputing devices. Specifically, aspects of the disclosure will bedescribed with regard to monitoring multiple requests by a clientcomputing device for an original resource and a set of correspondingembedded resources (collectively, a set of resources) and determiningdifferences in corresponding image data associated with rendered imagesof the set of resources. In one embodiment, an averaged reference imageassociated with the set of resources is determined and compared with asubsequently obtained sample image corresponding to the set ofresources. In another embodiment, the average reference image isweighted prior to comparing with the subsequently obtained sample image.Image differences can then be used to assess performance associated withrendering the set of resources. Additionally, based on this performance,an alert message can be provided to a content provider. Although variousaspects of the disclosure will be described with regard to illustrativeexamples and embodiments, one skilled in the art will appreciate thatthe disclosed embodiments and examples should not be construed aslimiting.

FIG. 1 is a block diagram illustrative of a Web site monitoring system100 for monitoring the performance associated with rendering Web sitecontent. As illustrated in FIG. 1, the Web site monitoring system 100includes a number of client computing devices 102 (generally referred toas clients) for requesting content from a content provider. Asillustrated in FIG. 1, each client computing device 102 includes aclient computing component 104 for requesting content from networkresources in the form of an originally requested resource that mayinclude identifiers to two or more embedded resources that need to berequested. Additionally, as will be further explained below, the clientcomputing device 102 may include an image capture component 106 thatgenerates an image of the content rendered by the client computingcomponent.

As illustrated in FIG. 1, the client computing component 104 and imagecapture component 106 are executed on each client computing device 102.In another embodiment, the client computing component 104 may functionin conjunction with a separate communication software application (e.g.,a browser software application) to provide the combined functionalitydescribed for the client computing component 104. For example, theclient computing component could correspond to a stand alone softwareapplication, plugin, script, and the like. Additionally, although eachclient computing device 102 is illustrated as having a separate imagecapture component 106, in an alternative embodiment, the image capturecomponent 106 may be shared by one or more client computing devices.

In an illustrative embodiment, the client computing devices 102 maycorrespond to a wide variety of computing devices including personalcomputing devices, laptop computing devices, hand-held computingdevices, terminal computing devices, mobile devices, wireless devices,various electronic devices and appliances and the like. As alsoillustrated in FIG. 1, the client computing devices 102 are consideredto be logically grouped, as represented generally by client 107,regardless of whether the client computing devices are physicallyseparate and geographically distributed throughout the communicationnetwork 114. In this regard, the client computing devices 102 may eachcommunicate directly or indirectly with other computing devices overnetwork 114, such as a wide area network or local network. Additionally,one skilled in the relevant art will appreciate that client 107 can beassociated with various additional computing devices/componentsincluding, but not limited to, content and resource administrativecomponents, DNS resolvers, scheduling devices/components, and the like.

Each of the client computing devices 102 can accordingly includenecessary hardware and software components for establishingcommunications over the network 114. For example, the client computingdevices 102 may include networking components and additional softwareapplications that facilitate communications via the Internet or anintranet. As previously described, the client computing device 102 mayinclude an additional, separate browser software application. The clientcomputing devices 102 may also be associated with, or otherwise include,other computing components, such as proxy applications, for furtherfacilitating communications via the Internet or an intranet. Aspreviously described, the client computing components 104 may eachfunction as a browser software application for requesting content from anetwork resource. In other embodiments, the client computing devices 102may be otherwise associated with an external proxy application, as wellas any other additional software applications or software services, usedin conjunction with requests for content.

With continued reference to FIG. 1 and as set forth generally above, theWeb site monitoring system 100 may include a content provider 108 incommunication with the one or more client computing devices 102 via thecommunication network 114. The content provider 108 may include a numberof content delivery components 110, such as a Web server component andassociated storage component corresponding to one or more servercomputing devices for obtaining and processing requests for content(such as Web pages) from the client computing devices 102. One skilledin the relevant art will appreciate that the content provider 108 caninclude or otherwise be associated with various additional computingresources, including, but not limited to, additional computing devicesfor administration of content and resources, DNS name servers,interfaces for obtaining externally provided content (e.g.,advertisements, Web services, etc.), and the like. Although the Web sitemonitoring system 100 is illustrated in a client-server configuration,one skilled in the relevant art will appreciate that the Web sitemonitoring system 100 may be implemented in a peer-to-peer configurationas well.

With yet further continued reference to FIG. 1, the Web site monitoringsystem 100 may further include a processing device 116 for analyzingimage data related to the processing of client requests. The processingdevice 116 can also be used to analyze the image data and to determineif an alert message and/or modifications to the original resource and/orembedded resources should be made to improve performance for subsequentclient requests for the corresponding content associated with theoriginal resource and/or embedded resources.

As illustrated in FIG. 1, the processing device 116 is in communicationwith the one or more client computing devices 102 and the contentprovider 108 via communication network 114. Additionally, as will befurther described below, the processing device 116 may include aprocessing component, such as an image analysis component 118, for theanalysis of image data from the client computing devices 102 and theassessment of whether to provide alert messages. The processing device116 may further include a local data store 120 for storing image data,such as averaged reference images, sample images, weight maps, and thelike, as will be further described below. It will be appreciated by oneskilled in the art and others that image analysis component 118 and datastore 120 may correspond to multiple devices/components and/or may bedistributed.

In further reference to FIG. 1, the Web site monitoring system 100 caninclude a number of service providers, other than the content provider108, for use in hosting, processing, or enabling the transmission ofcontent. Moreover, it will be appreciated by one skilled in the relevantart that a service provider may offer any one or more servicesassociated with CDN, storage, application, and transit serviceproviders. For example, a data center may provide both traditionalstorage services, as well as network computing services. For purposes ofsimplicity, the present description will simple reference content asbeing provided by the content provider 108.

One skilled in the relevant art will also appreciate that the componentsand configurations provided in FIG. 1 are illustrative in nature.Accordingly, additional or alternative components and/or configurations,especially regarding additional components, systems and subsystems forfacilitating communications may be utilized.

With reference now to FIGS. 2 and 3, an illustrative example of theoperation of the Web site monitoring system 100 according to someembodiments will be described. For purposes of the example, however, theillustration has been simplified such that many of the componentsutilized to facilitate communications are not shown. One skilled in therelevant art will appreciate that such components may be utilized andthat additional interactions would accordingly occur without departingfrom the spirit and scope of the present disclosure.

With reference to FIG. 2, a client computing component 104 initiates acontent request that is intended to ultimately be received and processedby the content provider 108. In an illustrative embodiment, therequested content may correspond to a Web page, or any portion thereof,that is displayed on the client computing device 102 via the processingof a base set of information, such as hypertext markup language(“HTML”), extensible markup language (“XML”), and the like. The base setof information may also include a number of embedded resourceidentifiers that corresponds to resource objects that should be obtainedby the client computing device 102 as part of the processing of therequested content. The embedded resource identifiers may be generallyreferred to as resource identifiers or resource URLs. The request forthe base set of information and the subsequent request(s) for anyembedded resources, or any portion thereof, may be referred to generallyas a “resource request.”

In one embodiment, prior to initiating a resource request, anothercomputing device, such as processing device 116, may be used todetermine whether a test to monitor performance associated withrendering a particular resource, such as a Web page, should beconducted. In this example, the processing device 116 may send the testrequest, which includes a resource identifier corresponding to thedesired resource request, to the client computing device 102.

In one illustrative embodiment, as shown in FIG. 2, the client computingcomponent 104 initiates the content request by transmitting the resourceidentifier to the content provider 108 via communication network 114.Thereafter, the content provider 108 receives the resource request fromthe client computing device 102 and processes the resource request usingcontent delivery components 110, such as a Web server. As shown in FIG.2, upon obtaining the requested resource, the content provider 108initiates transmission of the requested resource to the client computingdevice 102.

As further illustrated in FIG. 2, the client computing component 104obtains and processes the requested resource. For example, the clientcomputing component 104 can process the requested resource for displayon a monitor or other display device associated with the clientcomputing device 102. In addition or alternatively, the client computingcomponent 104 can process the requested resource for sending to anyother component or external device (e.g., a framebuffer). For example,as illustrated in FIG. 2, the client computing component 104 can processthe requested resource for sending to the image capture component 106.The image capture component 106 can then generate an image associatedwith rendering the requested resource. Such an image captures allvisible content and effectively normalizes it into one form of data.

It will be appreciated by one skilled in the art and others that theabove-described functionality of the image capture component 106 can beimplemented, for example, by a separate stand-alone component as part offunctionality provided by the operating system of the client computingdevice 102 (e.g., a driver loaded on the operating system), or acomponent being executed on a graphics card associated with the clientcomputing device for providing a screen image capture to a separatefile.

With reference now to FIG. 3, the image capture component 106 of theclient computing device 102 provides the generated image data associatedwith the requested resource to the image analysis component 118 of theprocessing device 116 via the communication network 114. The imageanalysis component 118 then processes the received image data.

In one embodiment, and as will be further described below, the receivedimage data can correspond to a first reference image associated with therequested resource. The image analysis component 118 can store the firstreference image in local data store 120, or any other data storedistributed across the network 114 for later use, such as after furtherreference images associated with the requested resource are obtained.For example, as similarly set forth above, prior to the client computingdevice 102 initiating a resource request, the processing device 116 cansend a test request, which includes a resource identifier correspondingto the desired resource request as well as the number of times theresource should be requested to generate separate sets of referenceimage data, to the client computing device 102. In accordance with thisembodiment, where the number of initial resource requests to be made bythe client computing device is two, the client computing device 102initiates two requests for the same resource and generates two sets ofreference image data as similarly described in reference to FIG. 2.Accordingly, in this example, the processing device 116 receives both afirst reference image and a second reference image from the imagecapture component 106. As will be further described below, the imageanalysis component 118 then determines an averaged reference image basedon the first and second reference images.

In some embodiments, the image analysis component 118 can then alsogenerate a weight map based on the first and second reference images.For example, as will also be described further below, the weight map caninclude a weighted pixel value for each pixel corresponding to therespective pixels in the first and second reference images. The imageanalysis component 118 can store the reference images and weight map inlocal data store 120, or any other data store distributed across thenetwork 114 for later use.

In accordance with another embodiment, and as will be further describedbelow, the generated image data received at the image analysis component118 from the image capture component 106 of the client computing deviceas shown in FIG. 3 can correspond to a sample image associated with therequested resource. A sample image associated with the requestedresource can be generated after the processing device has generated acorresponding averaged reference image. In one example, after theaveraged reference image is generated, the processing device 116 caninitiate a further test request for the client computing device 102 tocommence further resource requests for the same content in order togenerate sample image data that can be compared to the averagedreference image data. Alternatively, such further test request can be apart of the initial test request described above and may indicate a timefor subsequently initiating the requests to generate the sample imagedata.

As will be further described below, after receiving sample image datafrom the image capture component 106, the image analysis component 118of the processing device 118 can compare the sample image with thecorresponding averaged reference image to automatically determinewhether any content rendering errors or inconsistencies exist. Theprocessed sample image can accordingly be used to support modificationsto the original resource and/or embedded resources to improveperformance for subsequent client requests for the original resource. Ifnecessary, as will be further described below in reference to FIGS.6A-6C, the processing device 116 can cause the display of the averagedreference image, the sample image, and any inconsistencies,respectively, to a user of the system 100 for further assessment. Inaddition or alternatively, the image analysis component 118 can alsothen transmit an alert message to the content provider 108 identifyingany such errors or inconsistencies related to the processing of theclient request for the original resource and any embedded resources.

With reference now to FIG. 4, one embodiment of an image processingroutine 400 implemented by the Web site monitoring system 100 will bedescribed. One skilled in the relevant art will appreciate thatactions/steps outlined for routine 400 may be implemented by one or manycomputing devices/components that are associated with Web sitemonitoring system 100. Accordingly, routine 400 has been logicallyassociated as being generally performed by the Web site monitoringsystem 100, and thus the following illustrative embodiments should notbe construed as limiting.

At block 402, a client computing component 104 identifies a firstresource request associated with a first resource identifier andrequests the corresponding set of resources. As previously mentioned,the client computing component 104 can generate the first resourcerequest or receive the first resource request from another computingdevice, such as processing device 116. In one example, the firstresource request associated with the first resource identifier may befor a Web page, such as http://example.com. At block 404, an imagecapture component 106 of the client computing device 102 generates afirst reference image corresponding to a first rendering of the set ofresources associated with the first resource request. The firstreference image includes a pixel intensity value for each pixel in thefirst reference image. Essentially, the first reference image capturesall visible content and effectively normalizes into one form of data.

Next, at a block 406, a test is performed by the client computingcomponent 104 to determine whether another resource request associatedwith the first resource identifier should be initiated. The clientcomputing component can make this determination itself, based on newinstructions received from another computing device, such as processingdevice 116, or based on prior instructions from another computingdevice, such as processing device 116.

If the client computing component 104 determines that a second resourcerequest associated with the first resource identifier should beinitiated at block 406, processing returns to block 402 where theforegoing described functionality is repeated until no resource requestsare outstanding. In particular, the client computing component 102transmits a second resource request associated with the first resourceidentifier. In response, the image capture component 106 of the clientcomputing device 102 then generates a second reference imagecorresponding to a second rendering of the set of resourcescorresponding to the first resource identifier. The second referenceimage similarly includes a pixel intensity value for each pixel in thesecond reference image. Then, another test is made at block 406 todetermine whether yet another resource request associated with the firstresource identifier (e.g., uniform resource locator corresponding to aparticular Web page) should be initiated.

If at block 406, the client computing component 104 determines that noadditional resource requests should be initiated, processing continuesat block 408 where the image analysis component 118 of the processingdevice 116 determines an averaged reference image based on the priorreference images generated by and obtained from the client computingdevice 102. In one example, where the image analysis component obtainstwo reference images, the image analysis component 118 determines theaveraged reference image by averaging the pixel intensity value for eachpixel in the first reference image with the pixel intensity value foreach pixel in the second reference image.

Additionally, at block 408, the image analysis component 118 mayoptionally an image weight map based on the obtained reference images.The image weight map includes a weighted pixel value for each pixelcorresponding to the respective pixels in the reference images. Theimage analysis component 118 determines the weighted pixel values basedon the difference of the corresponding pixel intensity value between theobtained reference images. For example, the image analysis component 118may determine that a given pixel corresponds to static content if itremains constant between the obtained reference images and to moredynamic content if it does not. The image analysis component 118 mayweight static pixel content greater than dynamic pixel content. In oneembodiment, the weight map may be a binary weight map. In this instance,static pixel content may be given a value of 1, while dynamic pixelcontent may be given a value of 0. Although not illustrated in FIG. 4,the image analysis component 118 can store the averaged reference imageand any corresponding generated weight map in the data store 120 forfuture use as will be described below. The routine 400 ends at block410.

It will be appreciated by one skilled in the art and others that whilethe functionality associated with blocks 404 and 408 is described aboveas being implemented by the image capture component 106 of the clientcomputing device 102 and the image analysis component 118 of theprocessing device 116, respectively, the combined functionality may beimplemented by a single processing component of either the clientcomputing device 102 or the processing device 116. In addition, suchsingle processing component may also provide the instructions to theclient computing component regarding initiation of the resourcerequests.

With reference now to FIG. 5, one embodiment of an image monitoringroutine 500 implemented by the Web site monitoring system 100 will bedescribed. One skilled in the relevant art will appreciate thatactions/steps outlined for routine 500 may be implemented by one or manycomputing devices/components that are associated with the Web sitemonitoring system 100. Accordingly, routine 500 has been logicallyassociated as being generally performed by the Web site monitoringsystem 100, and thus the following illustrative embodiments should notbe construed as limiting.

At block 502, a client computing component 104 identifies anotherresource request associated with the first resource identifier andrequests the corresponding set of resources. As previously mentioned,the client computing component 104 can generate the additional resourcerequest or receive the additional resource request from anothercomputing device, such as processing device 116. In one embodiment, thisadditional resource request is generated after the Web site monitoringsystem has generated the averaged reference image associated with thefirst resource identifier as described in reference to FIG. 4. At block504, an image capture component 106 of the client computing device 102generates a sample image corresponding to an additional rendering of theset of resources associated with the additional resource request ofblock 502. Similar to the reference images, the sample image includes apixel intensity value for each pixel in the first reference image.Essentially, the sample image also captures all visible content andeffectively normalizes into one form of data.

Continuing at block 506, the image analysis component 118 of theprocessing device 116 processes the sample image generated by andobtained from the client computing device 102. In particular, the imageanalysis component 118 compares the sample image with the averagedreference image corresponding to the same set of requested resources inorder to identify a compared image. In one embodiment, the resultingcompared image includes an intensity value for each pixel of thecompared image corresponding to the difference between the correspondingpixel in sample image and the averaged reference image. Accordingly, inone illustrative embodiment, pixels in the compared image with a valueof 0 or a value which does not exceed an individual pixel thresholdvalue are determined to have remained essentially static. However,pixels with a value exceeding the individual pixel threshold value aredetermined to be dynamic.

Next, at block 508, the image analysis component 118 automaticallydetermines whether any content rendering errors or inconsistenciesbetween the sample image and the corresponding averaged reference imageexist. In one embodiment, the image analysis component 118 makes such adetermination by first weighting the compared image using thecorresponding weight map generated as described in reference to FIG. 4.Accordingly, each pixel value of the compared image is multiplied by thecorresponding pixel value of the weight map. Next, the image analysiscomponent 118 determines whether a sum of the intensity values for eachpixel in the weighted compared image exceeds a threshold. In thisexample, if the threshold is exceeded, the image analysis component 118determines that content rendering errors or inconsistencies between thesample image and the corresponding averaged reference image in factexist.

Continuing at block 510, the image analysis component 118 then transmitsan image analysis message if necessary. In one embodiment, an imageanalysis message can be transmitted to the content provider 108responsible for the set of resources associated with the analyzed imageswhether or not the image analysis component 118 determines that errorsor inconsistencies exist. In this example, the image analysis messagesimple reports all results to the content provider 108.

In another embodiment, the image analysis message can be transmitted tothe content provider only in the event that the image analysis component118 determines that errors or inconsistencies exist as determined atblock 508. In this example, the image analysis message can be an alertmessage. The alert message can include an identification of an affectedpixel area corresponding to the rendered set of resources so that acontent provider can identify the underlying affected content. Theaffected area can be determined by the image analysis component 118 asthe pixel area in which the greatest differences in pixel intensityvalue exist between the sample image and the averaged reference image.In addition or alternatively, the alert message can include anidentification of an expected output for the affected area. In oneexample, this identification may be a visual display of the expectedcontent for the affected area within a bounding box defining theaffected area as illustrated in FIG. 6C. In addition or alternatively,the alert message can include an identification of a responsible entityfor the content in the affected area. Moreover, the alert message may besent directly to this responsible entity. Even further, the alertmessage can include other data corresponding to the rendered imagesassociated with the set of resources, such as an identification of theWeb browser used to render the images.

In another embodiment the image analysis message or alert message mayadditionally or alternatively be transmitted to an administrator of theWeb site monitoring system for further action. The routine 500 ends atblock 512.

As similarly set forth above in reference to FIG. 4, it will beappreciated by one skilled in the art and others that while thefunctionality associated with blocks 504 and 506-510 is described aboveas being implemented by the image capture component 106 of the clientcomputing device 102 and the image analysis component 118 of theprocessing device 116, respectively, the combined functionality may beimplemented by a single processing component of either the clientcomputing device 102 or the processing device 116. In addition, suchsingle processing component may also provide the instructions to theclient computing component regarding initiation of the resourcerequests.

With reference now to FIG. 6A, an illustrative user interface 600Agenerated by the processing device 116 for displaying an expected imagecorresponding to the set of resources (e.g., the content on a Web page)is provided. In one embodiment, the expected image corresponds to theaveraged reference image. This user interface 600A can be provided to acontent provider 108 as a part of an image analysis message or an alertmessage as described above. As will also be discussed further below, theuser interface 600A includes a current event section 602A and a buybutton 604A.

With reference now to FIG. 6B, an illustrative user interface 600Bgenerated by the processing device 116 for displaying a sample imagecorresponding to the set of resources (e.g., the content on a Web page)and compared against the corresponding averaged reference image isprovided. This user interface 600B can be provided to a content provider108 as a part of an image analysis message or an alert message asdescribed above. In particular, it can be provided in conjunction withthe user interface 600A as described in reference with FIG. 6A toprovide a comparison to the content provider of how a Web page iscurrently rendering versus how is should be rendering. As shown in FIG.6B, the sample image displayed in the user interface 600B has adifferent current event identified in current event section 602B. Inthis example, an upcoming cross-country cycling event in shown incurrent event section 602B, as compared to a previous bike sale as shownin current event section 602A in FIG. 6A. In addition, the sample imagedisplayed in the user interface 600B is missing the buy button aspreviously displayed in FIG. 6A.

With reference now to FIG. 6C, an illustrative user interface 600Cgenerated by the processing device 116 for displaying an affected imagearea corresponding to the set of resources (e.g., the content on a Webpage) in which errors or inconsistencies have been found is provided.The user interface 600C illustrates a visual display of the content inthe affected pixel area only. The content displayed within the boundingbox defining the affected area can illustrate the expected content forthat area. This user interface 600C can be provided to a contentprovider 108 as a part of an image analysis message or an alert messageas described above. Moreover, the user interface 600C can be providedalone, or together with user interfaces 600A and 600B for furtherclarity.

Continuing with the foregoing example, two differences were found in acomparison of the averaged reference image in FIG. 6A and the sampleimage in FIG. 6B. One was a difference between the current eventsections of the user interfaces, and the other difference was that thebuy button was missing from the user interface in FIG. 6B. As shown inFIG. 6C, the only affected area shown is associated with the buy button.In this example, a weight map applied to an image comparison between theaveraged reference image and the sample image indicated that the currentevent section in the reference images used to generate the averagedreference image were constantly changing over time. Accordingly, theweight map accorded a zero weight to the current event section.Accordingly, when applied to the image comparison between the averagedreference image and the sample image, the current event section was notidentified as an affected area as it is expected to constantly change,whereas the buy button should remain static.

In another embodiment, the image tested through generation of referenceimages and one or more sample images may correspond only to the buybutton area of the Web page or any other known static area ofimportance, rather than the entire content of the Web page.

It will be appreciated by those skilled in the art and others that whileprocessing, monitoring, and other functions have been described hereinas being performed at various components of the client computing device102 and/or the processing device 116, these functions can be distributedacross one or more computing devices. In addition, the image data andweight maps generated at the client computing device 102 and/or theprocessing device 116 can be maintained globally by the processingdevice 116 and shared with all or some subset of the components of theclient computing device 102.

It will further be appreciated by those skilled in the art and othersthat all of the functions described in this disclosure may be embodiedin software executed by one or more processors of the disclosedcomponents. The software may be persistently stored in any type ofnon-volatile storage.

Conditional language, such as, among others, “can,” “could,” “might,” or“may,” unless specifically stated otherwise, or otherwise understoodwithin the context as used, is generally intended to convey that certainembodiments include, while other embodiments do not include, certainfeatures, elements and/or steps. Thus, such conditional language is notgenerally intended to imply that features, elements and/or steps are inany way required for one or more embodiments or that one or moreembodiments necessarily include logic for deciding, with or without userinput or prompting, whether these features, elements and/or steps areincluded or are to be performed in any particular embodiment.

Any process descriptions, elements, or blocks in the flow diagramsdescribed herein and/or depicted in the attached figures should beunderstood as potentially representing modules, segments, or portions ofcode which include one or more executable instructions for implementingspecific logical functions or steps in the process. Alternateimplementations are included within the scope of the embodimentsdescribed herein in which elements or functions may be deleted, executedout of order from that shown or discussed, including substantiallyconcurrently or in reverse order, depending on the functionalityinvolved, as would be understood by those skilled in the art.

It should be emphasized that many variations and modifications may bemade to the above-described embodiments, the elements of which are to beunderstood as being among other acceptable examples. All suchmodifications and variations are intended to be included herein withinthe scope of this disclosure and protected by the following claims.

What is claimed is:
 1. A computer-implemented method comprising:generating a first reference image corresponding to a first rendering ofat least a portion of a set of resources, wherein the first referenceimage comprises a pixel intensity value for individual pixels in thefirst reference image; generating a second reference image correspondingto a second rendering of the same at least a portion of the set ofresources in the first rendering, wherein the second reference imagecomprises a pixel intensity value for individual pixels in the secondreference image; and determining an image weight map including aweighted pixel value for individual pixels corresponding to therespective pixels in the first and second reference images, whereinindividual weighted pixel values are based on the difference of thecorresponding pixel intensity value between the first and secondreference images, wherein the method is implemented by a computer systemconfigured with specific executable instructions.
 2. Thecomputer-implemented method as recited in claim 1, wherein the set ofresources corresponds to Web page content.
 3. The computer-implementedmethod as recited in claim 1, wherein the first and second referenceimages are individual representations of a rendering of a portion of theWeb page content.
 4. The computer-implemented method as recited in claim1, wherein the image weight map is a binary weight map.
 5. Thecomputer-implemented method as recited in claim 1, wherein the imageweight map includes a plurality of weighted pixel values and whereinindividual weighted pixel values in the weight map are based on whetherthe corresponding pixel intensity value in the first and secondreference images is constant.
 6. The computer-implemented method asrecited in claim 1 further comprising: transmitting a third resourcerequest associated with the first resource identifier, wherein the thirdresource request corresponds to the same set of resources requested inthe first and second resource requests; and generating a sample imagecorresponding to a third rendering of the same at least a portion of theset of resources in the first rendering, wherein the sample imagecomprises a pixel intensity value for individual pixels in the sampleimage.
 7. The computer-implemented method as recited in claim 6 furthercomprising: comparing the sample image with an averaged reference imageto identify a compared image, wherein the averaged reference image isdetermined by averaging pixel intensity values in the first referenceimage with pixel intensity values in the second reference image;weighting the compared image using the image weight map to produce aweighted compared image; and determining whether a sum of the intensityvalues for individual pixels in the weighted compared image exceeds athreshold.
 8. The computer-implemented method as recited in claim 7,wherein the compared image includes an intensity value for individualpixels of the compared image corresponding to the difference between thesample image and the averaged reference image.
 9. Thecomputer-implemented method as recited in claim 7 further comprising:transmitting an alert message if the threshold is exceeded.
 10. A systemcomprising: at least one computing device having specific executableinstructions stored in an associated memory and operative to: generate afirst reference image corresponding to a first rendering of at least aportion of a set of resources, wherein the first reference imagecomprises a pixel intensity value for individual pixels in the firstreference image; generate a second reference image corresponding to asecond rendering of the same at least a portion of the set of resourcesin the first rendering, wherein the second reference image comprises apixel intensity value for individual pixels in the second referenceimage; and determine an image weight map including a weighted pixelvalue for individual pixels corresponding to the respective pixels inthe first and second reference images, wherein individual weighted pixelvalues are based on the difference of the corresponding pixel intensityvalue between the first and second reference images.
 11. The system asrecited in claim 10, wherein the set of resources corresponds to Webpage content.
 12. The system as recited in claim 10, wherein the firstand second reference images are individual representations of arendering of a portion of the Web page content.
 13. The system asrecited in claim 10, wherein the image weight map is a binary weightmap.
 14. The system as recited in claim 10, wherein the image weight mapincludes a plurality of weighted pixel values and wherein individualweighted pixel values in the weight map are based on whether thecorresponding pixel intensity value in the first and second referenceimages is constant.
 15. The system as recited in claim 10, wherein theat least one computing device is further operative to: obtain a thirdresource request associated with the first resource identifier, whereinthe third resource request corresponds to the same set of resourcesrequested in the first and second resource requests; and generate asample image corresponding to a third rendering of the same at least aportion of the set of resources in the first rendering, wherein thesample image comprises a pixel intensity value for individual pixels inthe sample image.
 16. The system as recited in claim 15, wherein the atleast one computing device is further operative to: compare the sampleimage with an averaged reference image to identify a compared image,wherein the averaged reference image is determined by averaging pixelintensity values in the first reference image with pixel intensityvalues in the second reference image; weight the compared image usingthe image weight map to produce a weighted compared image; and determinewhether a sum of the intensity values for individual pixels in theweighted compared image exceeds a threshold.
 17. The system as recitedin claim 16, the compared image includes an intensity value forindividual pixels of the compared image corresponding to the differencebetween the sample image and the averaged reference image.
 18. Thesystem as recited in claim 16, wherein the at least one computing deviceis further operative to: provide an alert message if the threshold isexceeded.
 19. Computer-readable non-transitory storage includingcomputer-executable instructions that, when executed by a processor,cause the processor to at least: generate a first reference imagecorresponding to a first rendering of at least a portion of a set ofresources, wherein the first reference image comprises a pixel intensityvalue for individual pixels in the first reference image; generate asecond reference image corresponding to a second rendering of the sameat least a portion of the set of resources in the first rendering,wherein the second reference image comprises a pixel intensity value forindividual pixels in the second reference image; and determine an imageweight map including a weighted pixel value for individual pixelscorresponding to the respective pixels in the first and second referenceimages, wherein individual weighted pixel values are based on thedifference of the corresponding pixel intensity value between the firstand second reference images.
 20. The computer-readable non-transitorystorage as recited in claim 19, wherein the set of resources correspondsto Web page content.
 21. The computer-readable non-transitory storage asrecited in claim 20, wherein the first and second reference images areindividual representations of a rendering of a portion of the Web pagecontent.